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TITLE OF THE INVENTION 

A LAYER EDITOR SYSTEM FOR A PEN-BASED COMPUTER 
CROSS-REFERENCE TO RELATED APPLICATION(S) 

[0001] This application is related to and claims priority to U.S. provisional application 
entitled "User Interface Techniques For Pen-Based Computers" having serial number 
60/438,499, by Fitzmaurice et al, filed January 8, 2003; this application is related to U.S. 
application entitled "A User Interface Having A Placement And Layout Suitable For Pen-Based 
Computers" having serial number **(1 252.1 087), by Fitzmaurice et al, filed concurrently 
herewith; and this application is related to U.S. application entitled "Biomechanical User 
Interface Elements For Pen-Based Computers" having serial number **(1 252. 1088), by 
Fitzmaurice et al, filed concurrently herewith; and all of which are incorporated by reference 
herein. 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

[0002] The present invention is directed to a layer editor for a pen-based computer and, 
more particularly, to system where user identification of layers does not require text input, 
commands can be applied to the layers using a marking menu approach and where the 
invocation of the marking menus is facilitated by large "hit areas" for the menus in the layer 
editor. 

2. Description of the Related Art 

[0003] Pen or stylus based computers, such as tablet personal computers (PCs) and 
personal digital assistants (PDAs) are becoming popular. These computers typically allow the 
user to interact with the computer through a graphical user interface using a stylus or pen (or 
possibly even a finger). A keyboard and a mouse are typically not available. Pen-based type 
interaction can be limiting as compared to other types of interaction, such as using the mouse or 
the keyboard, alone or in combination. What is needed are improvements to the interaction 
capability of pen-based computers. 

[0004] User interfaces - Ul (layer editors) providing access to the layer functionality of paint 
systems are designed for mouse based user interfaces. Layer editors provide a representation 
of each paint layer that can be manipulated or edited. Typical examples are the layer editor in 
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PhotoShop or the layer editor in "Fractal paint". These types of Ul present difficulties when 
operated on pen-based computers. Specifically, the naming of layers in a traditional Ul requires 
text input and this is hard to do with a pen. Also, buttons and widgets on a traditional Ul can be 
too small to accurately hit with a pen. What is needed is a system that will allow writing based 
naming of layers and controls that can be more easily targeted with a pen or stylus. 

[0005] In addition, most keyboard and mouse based user layer editor interfaces use hot- 
keys to allow a user to quickly access layer commands that are normally in menu-items on the 
menu bar or on pop-up menus. With a pen based user interface no keyboard may be available 
or the user may prefer not to use the keyboard. What is needed is to provide pen-based 
computers fast access capability somewhat equivalent to hot key capability. 

SUMMARY OF THE INVENTION 

[0006] It is an aspect of the present invention to provide a layer editor where user 
identification of layers does not require text input. 

[0007] It is another aspect of the present invention to allow commands to be applied to the 
layers using marking menu technology. 

[0008] It is an additional aspect of the present invention to provide large hit areas or zones 
for the controls of the layers. 

[0009] The above aspects can be attained by a system that provides a layer editor 
representing drawing layers using box like controls where the name of each layer depicted in 
the box for that layer can be created using writing strokes of a stylus or pen of a pen-based 
computer. The pen is also used to select the controls, pop-up menus and perform marking 
selections or operations with underlying menus or functions. A drawing dialog is used to write 
the names applied to the layers. 

[0010] These together with other aspects and advantages which will be subsequently 
apparent, reside in the details of construction and operation as more fully hereinafter described 
and claimed, reference being had to the accompanying drawings forming a part hereof, wherein 
like numerals refer to like parts throughout. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0011] Figure 1 illustrates a layer editor user interface (Ul) of the present invention. 
[0012] Figure 2 depicts marking menus of the layer editor of the present invention. 
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[0013] Figure 3 depicts a drawing dialog box of the present invention used for naming the 
layers of the layer editor of the present invention. 

[0014] Figure 4 depicts the hardware of the present invention. 

[0015] Figure 5 shows the operations of the present invention. 

[0016] Figure 6 illustrates the preferred data structure of the present invention. 

[0017] Figures 7 - 13 depict the components of the Ul in more detail. 

[0018] Figures 14-21 illustrate additional embodiments of the layer editor of the present 
invention. 

[0019] Figure 22 illustrates using the present invention for a slide presentation editor. 
[0020] Figures 23 - 25 depict using gestures for layer editing. 
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0021] The present invention is directed to a layer editor system that enables users to avoid 
text input by drawing the names of layers by hand with the pen on the layers represented in the 
layer editor. The invention also provides larger hit zones or areas so button/control operation 
with a pen is easier. Command engagement is further facilitated by the use of marking menus 
on each layer representation to make command engagement very easy with a pen. 

[0022] The layer editor 1 0 of the present invention, as depicted in figure 1 , can be displayed 
on the display screen of a pen-based or other type of computer using any common method of 
command invocation. Typically this is done from a "Layers Editor" menu-item in the menu bar. 
Figure 1 shows an example where the layers editor 10 is displayed by selecting, for example, 
Edit-> Layers in the menu-bar or pop-up menu 8. Invoking the layer editor 10 can be done in 
many different ways, such as with a special button on the screen, or by pressing a key on the 
keyboard. 

[0023] The layer editor 10 is a graphical user interface that preferably includes a box like 
graphic 12 that encloses the different controls, etc. of the editor 10. To make the layer editor 10 
easier to use with a pen or stylus, the controls in the layer editor 10 should be relatively large or 
present relatively large targets for the pen user to hit. Large controls, in the context of a pen or 
stylus, relate to the accuracy of the pointing device. Typically, pointing with a pen moves the 
cursor to that location +/- e, where e is the distance "error accuracy of the pen". Preferably, a 
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target has to be at least 2e in size to assure that if the user does point to the center of the target 
the cursor will be within the target. With current tablet pc pens or styli e is about 1 -2mm, so 
targets are preferably at least 4 mm high and wide. This of course is an extreme minimum: 
10mm is much more acceptable and preferable. In addition, "Fitts Law" of pointing indicates 
that the ease of pointing is proportional to the width of the target, so bigger is generally better. 
As a result, we prefer that a layer editor button or control have a target size of about 20 mm 
(10e) for minimum height and width. 

[0024] When the layer editor 10 is displayed it preferably has many of the common window 
features typically found in modern graphical user interfaces. These features include a title bar 
14 that allows a user to invoke a layers dialog. A minimize button or control 16 that allows the 
editor 10 to be minimized. A close button 18 closes the editor 10. A scrollbar 20 allows the 
different layers depicted by the editor 10 to be displayed where the typical editor displays five 
layers and can be scrolled to display all of the layers. These controls may be enlarged as 
needed to allow easier operation with a pen. 

[0025] The present invention provides for the use of boxes displayed to represent the 
ordering of the drawing layers in a paint application. Figure 1 shows a bottom to top ordering of 
four layers 22-28 where the background layer 28, hand letter named "ground", is at the bottom 
of the layer editor and subsequent layers are "layered on top over 1 ' each other. This vertical 
ordering represents the underlying ordering of the data structure, such as a pointer directed list 
implementing a layer stack, that maintains the layers for the particular drawing. The stack 
entries also contain the parameter data for each of the layers, such as locked/unlocked status, 
hidden/visible status, layer name bitmap, active, etc. Each layer can be a control that allows a 
function to be performed, such as display the layer in a one-shot type function or display a menu 
of operations that can be performed on the layer in a menu access type multiple operation type 
function. Each layer box displays its name and some state information about the layer (a layer 
may be hidden or visible, selected or unselected, locked, etc.). For example, layer 24 is the 
named the "body" layer and is a hidden layer indicated by using a gray overlay to indicate the 
hidden status. Layer 22 is named the "windows" layer and is also a hidden layer, but it is locked 
as depicted by the lock icon 30. This icon 30 is typically not a control because it is a very small 
target; however, the icon 30 could be a control providing a toggle between locked and unlocked. 
A locked layer cannot be moved. The background layer 28 is a visible layer, as indicated by the 
transparent overlay indicator and is also locked. The background layer 28 also preferably has a 
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visual indicator 32, in this case the text "BACKGROUND", indicating that the layer is the 
background layer. Layer 26 is named the "wheels" layer. Layer 26 is a visible layer. Layer 26 
is shown as having been selected by the user by the selection indication frame graphic 32 that 
has been placed around the layer box 26, thereby highlighting box 26. When a layer is 
selected, controls specifically for that layer are also presentable. Figure 1 shows a layer move 
control 36 that allows the layer to be moved up or down in the editor layer stack. A 
transparency control 38 for the layer allows the user to set the transparency degree for the 
layer, in this case layer 26. These controls 36 and 38 appear when the layer is selected. 

[0026] As noted above, the status icon 30 is preferably not a control. A reason that it is 
preferred that the lock icon 30 not be a control is the desire to make the target area for the layer 
control large for easy targeting and activation with a stylus. As a result, the present invention 
makes the menu target areas large by overlapping status indicators (like the lock icon 30) with 
the menu target area. In the present invention rather than having the status change or a status 
related function be activated when the icon is clicked, independent of the function of menu 
target area (for example, clicking on the lock unlocks the layer, preferably, clicking on the status 
icon pop-ups of the marking menu of the menu target area. In this way the menu target area 
includes the status areas and hence is larger and easier to hit. 

[0027] A particular layer can be selected by tapping on it with the pen, which can result in a 
visual indication on the layer box, such as the highlight frame, that it is selected. This selection 
is conventionally performed by comparing the position of the cursor/pen to a map of the control 
hit zones for the editor 10 and detecting a selection and a corresponding control when the 
cursor is within a particular layer or control hit zone of the map. The layer box is divided into 
three zones, the hit zones for and coincident with controls 36 and 38 and zone 40 
corresponding to and activating a pop-up marking type menu 50 (see figure 2 and the 
discussion below). 

[0028] To facilitate fast access to multiple functions for each layer, each layer box (or 
control) preferably has a pop-up menu 50 associated with it as shown in figure 2. To pop-up this 
menu 50, preferably the user has to press and hold the pen on a layer box and this pops-up a 
menu of commands that can be applied to the pressed on a layer. Note that press and hold first 
selects the layer box so a user does not have to separately click to select the layer, then with 
the press and hold selection of a particular layer activated, the application or execution/initiation 
of a command can be done with a single drag action (a mark) through the desired item of the 
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menu, which is a conventional marking menu type selection with the menu visible (or invisible 
as desired). As discussed above, an aspect of this invention is that the menus can be 
implemented as conventional "marking menus" where activation of one of the controls via a 
mark simultaneously selects a corresponding layer and selects an operation on the layer. Other 
types of pop-up menus could be used, such a traditional linear menus or "pie menus", however, 
marking menus are particularly suited to pen-based applications because a selection can be 
made very quickly by a simple straight stroke with the pen in the direction of the desired menu 
item. 

[0029] Figure 2 also depicts a preferred pop-up, drag operation selection menu 50 open 
over two layer editors. Each of the menus 50 is popped up for the active layer selected by the 
user. This menu has radial menu selections including a new layer selection 52 that inserts a 
new layer above or preferably below the selected or active layer. A clear layer control 54 that 
conventionally causes the system to clear the contents of the active layer. A rename layer 
selection 56 that allows the user to rename the layer using gestures or strokes of the pen/stylus 
to write the new name into the layer box, as will be discussed in more detail later. A delete layer 
menu button 58 conventionally deletes the current active layer by removing it from the stack 
when activated and the layer immediately below the deleted layer becomes the active layer. 
The merge control 60 causes the system to conventionally combine or compose the active layer 
with the layer below the active layer so that the content of the layers are combined. A lock layer 
selection 62 allows the user specify that the system lock/unlock the active layer. A hide layer 
selection 64 causes the active layer to be converted by the system to/from being a hidden layer 
when activated. The position layer control 66 puts the system into a mode in which the 
dragging on the layer itself (i.e., the cursor turns into a "dragging hand icon" and the user can 
drag the cursor on the image) moves the currently selected layer in 2 dimensions relative to the 
rest of the image. Note this is different from changing the ordering of the layers using the layer 
move control 36. This position function is exited by selecting another tool, such as a pencil 
drawing tool. 

[0030] Another feature of the invention is that the name contents of each layer box can be 
hand-drawn by the user. As shown in figure 3, when a user selects the "Rename Layer 1 ' menu 
item 56 of the marking menu 50, a conventional drawing dialog box 80 is displayed or pops-up. 
The dialog box 80 includes a drawing zone or area 82 where the user can simply draw any type 
of graphics to identify the layer. The dialog box 80 also includes controls or buttons for clear 84 
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that causes the system to clear the contents of the area 82, cancel 84 that causes the dialog 
box to be canceled by the system and OK 88. Pressing the OK control 88 causes the system to 
un-display the dialog box 80, store the hand drawn graphic of the drawing area contents in the 
layer editor data structure and place the hand drawn graphic into the corresponding layer box. 

[0031] The drawing area 82 of the dialog box 80 can be conventionally configured to 
support all drawing operations such as different colors, pen brushes, and visual effects found in 
paint and draw programs. The user input to the drawing area does need to be hand-drawn only. 
A visual reduction of the corresponding actual drawing layer can conventionally be provided to 
identify the layer (supplied by the system or the user). Alternatively, any small image could be 
input or a large image input and conventionally reduced by the system to identify the layer. One 
variation of this embodiment is to use a special menu item on the marking menu, entitled "make 
thumbnail," to allow a user to create a layer box image by conventionally copying and reducing 
the image associated with a layer into the layer box. The image for the layer box can also be 
created by a "snapshot" style of interaction. For example, the "Rename Layer" dialog box 80 
can have an addition button labeled "snapshot" that, when clicked, allows the user to select 
some portion of the screen to be used as the image for the layer box. The source of an image 
for a layer can even be a video or any dynamic visual element. The input to label a layer could 
also be in the form of "ink strokes" as specified by the Microsoft Tablet PC specification. This 
allows other types of commands, such as "typeface" or "bold", to be conventionally applied to 
the hand-drawn strokes. Speech recognition or character recognition for the drawn strokes 
could also be implemented to provide a textual string used to name the layer. Different types of 
input could be combined into the "Rename Label" dialog image. For example, the user may 
paste an image into the dialog box and then write on top of that image. Another variation of the 
"Rename Layer" functionality is to have the layer box be directly drawn into (meaning that the 
rename commands and dialog box would not needed). In a further variation embodiment, the 
hand drawn area of the layer box could also pop-up and be separated from the area that has a 
pop-up menu associated with it. 

[0032] The present invention preferably uses a hardware platform 100, as depicted in figure 
4. This platform 100 includes a conventional storage 102 (RAM, ROM, hard disc, floppy disc, 
CD, DVD, etc.) for storing, paint programs, data structures, data, etc. used by the present 
invention as well as the processes, bitmap graphics and bitmap hit zones of the invention 
discussed herein. The platform 100 also includes a conventional processor 104, typically a 
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tablet personal computer (Tablet PC) class machine. The platform 100 further includes a 
conventional pen based I/O system 106, such as a display with a pen based digitizer, and 
optionally includes conventional I/O components, such as a key board, floppy disc, mouse, etc. 

[0033] The operations of the process for the pen based layer editor of the present invention 
are depicted in figure 5. When the layer editor is invoked 120, the system creates 122 a layer 
display from a list of the layers in the paint system. This involves determining the number of 
layers and producing a layer editor graphic with the number of boxes needed and applying it to 
the display at a desired or predetermined location, obtaining the graphics for the layer names 
from storage and applying them to the corresponding boxes, obtaining the graphic for the other 
controls such as the minimize button and applying them to the display at corresponding 
locations, determining whether a hidden layer overlay or locked icon needs to be applied, etc. 
Then each layer is activated 124 as the corresponding controls so that a user can "click-on" 
each layer or on the other buttons to perform an associated function or display an associated 
pop-up display. The system then waits 1 26 for one of the layer controls to be activated. If a 
layer box control has been activated as discussed above, the system selects 128 or activates 
the layer under the pen cursor and provides the box on the display for the activated layer with 
the frame visually indicating that the layer has been activated. The system them determines the 
type of control that has been activated. This includes determining 130 whether the control is the 
transparency control 38 or the reorder control 36. If one of these controls have been activated, 
the system pops up the appropriate control and performs the called for interaction 132. If 
theses controls are not selected, the system determines 134 whether the user is making a 
marking or radial menu type selection of items in the layer operation menu 50. If so, the menu 
is popped up and/or a selection is made and the system performs the selected function. 

[0034] The preferred data structure of the present invention is depicted in figure 6. As 
depicted linked data structure includes: 

a Layer Editor 140 includes: 

a List of LayerEditorBoxes 142 

a Scroll Control 144 
a LayEditorBox 146 includes: 

Layer Image 148 

Layer Box Image 150 

Popup-menu 152 

Popup menu target zone 154 

Move layer control 1 56 

Move layer control target zone 1 58 
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Transparency control 160 
Transparency control target zone 162 

[0035] Figures 7-1 3 illustrate the controls of the layer editor 1 0 of the present invention in 
more detail. Figure 7 depicts the layer editor 500 with two layers 1 64 and 1 66 "New" and 
"Background". In this example, New 166 is the active layer as indicated by the border. This 
layer has three control zones: marking menu commands 168; re-order layers 170; and layer 
opacity 172 controls. Figure 8 shows the re-order layers transparent control visual 170 with 
both up 174 and down 176 controls for moving the active layer respectively within the layer 
stack. Figure 9 depicts the layer opacity control 172 in more detail. Selecting control 172 brings 
up a dialog box 178 depicted in figure 10. The layer opacity dialog box 178 includes a slider 
180 for setting the opacity of the layer from completely opaque to completely transparent. 
Figure 11 illustrates the commands of the marking menu control 182 for each layer with the 
preferred icons. These layer marking menu commands (starting North and going clockwise) 
include Add new layer icon/control 184 depicting a layer with a "+", Delete layer icon/control 186 
depicting a layer with a Rename layer icon/control 188 depicting a layer with lettering, 
Flatten image icon/control 190 depicting layers being pushed together by arrows, Merge with 
Below icon/control 192 depicting two layers and a merge arrow, Lock layer icon/control 194 
depicting a layer and a lock, Hide/Show layer icon/control 196 depicting a layer with an eye, and 
Move layer icon/control 198 depicting a grab capable, open hand with a layer, the functions of 
the controls are discussed above with respect to figure 2. Figure 12 depicts the reordering of 
the layer stack by a user selecting the transparent "re-order" graphic 170 on the active layer and 
dragging to the new position with the drag icon (closed grab hand) being depicted. Figure 13 
depicts the New Layer dialog box 200 in which the user can draw in a textual name or sketch a 
graphic using a pen or stylus and this will be the new image associated to the active layer. 

[0036] The layer editor interface as discussed above represents the layers as boxes 
arranged vertically. Other arrangements or embodiments are possible. Figure 14 illustrates an 
arrangement of horizontal connected boxes 212. Figure 15 depicts tiled overlapping boxes 214. 
Figure 16 shows non-connected trapezoid type boxes 216. Figure 17 illustrates an alternate 
embodiment of circular layer controls 218. Figure 18 illustrates a mixed numerically labeled and 
overlapping display order of variously shaped layer representation controls 220. Figure 19 
depicts controls 222 with no spatial ordering where each box is displayed one at a time, possibly 
in order of time of creation. The layer editor controls can also be graphically "attached" to the 
display of the paint layers themselves as depicted in figure 20. Figure 20 shows a list of tabs 
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224 along the display edge of the painted layered image 226 where the layer box (the tab in this 
case) is visually connected to the actual image. 

[0037] Another embodiment, as depicted in figure 21 provides layer names 252 and pop-up 
menus 254 on the layers themselves. Figure 22 is an image where three semi-transparent 
layers are placed over another image and on each layer a handwritten name or drawing is used. 
Each layer also has a spot where the pop up menu can be activated (the pop-up menu is shown 
with numbered menu items but these items could be typical layer commands such as: new 
layer, clear layer, rename layer etc.). 

[0038] Ultimately any ordered list or array could be used. Furthermore this invention is 
applicable to other areas beyond layer editors. For example, the list or array could be the 
thumbnails for a set or presentation slides 266 which can be hand labeled by the user and 
controlled by marking menus (267) on each slide as depicted in figure 22. A 2d array of 
thumbnail images for a slide presentation could also be used. The slides are labeled by hand 
drawn graphics and a marking menu is used to apply commands to the slide boxes. 

[0039] The user can also interact with the layer editor by using gestures and conventional 
gesture recognition and command invocation rather than menus. For example, as shown in 
figure 23, rather than popping up a menu on a layer and selecting the rename command to 
invoke the rename dialog, a user can invoke the "Rename" dialog by writing an "R" symbol over 
the intended layer 278. Gestures that span several layer boxes can also be use to specify 
several layer boxes simultaneously. For example, as shown in figure 24, the letter "M" drawn 
between two layer boxes 280 and 282 can be used to cause the system to merge the two layers 
into one layer. The gestures do not necessarily have to have "ink trails" which are the sequence 
of pixels activated when a pen stroke is made. Also, a gesture can span outside of a layer box, 
perhaps with some point (typically the starting point) in/on the layer control 284 as depicted in 
figure 25. A gesture could also be a scratch-out-mark 286 to indicate clearing or deleting of 
layers as shown in figure 26. 

[0040] As can be seen from the above discussion, the present invention has several 
advantages. First, user identification of layers does not require text input. This is extremely 
important for a pen-based computer especially when no keyboard is available or keyboard or 
text entry is cumbersome. Second, commands can be applied to the various layers very quickly 
and easily using marking menu technology. Third, the invocation of the marking menus is 
facilitated by large "hit areas" for the menus. 
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[0041] The present invention can also be used with mouse or other I/O device systems. 

[0042] The system also includes permanent or removable storage, such as magnetic and 
optical discs, RAM, ROM, etc. on which the process and data structures of the present invention 
can be stored and distributed. The processes can also be distributed via, for example, 
downloading over a network such as the Internet. 

[0043] The many features and advantages of the invention are apparent from the detailed 
specification and, thus, it is intended by the appended claims to cover all such features and 
advantages of the invention that fall within the true spirit and scope of the invention. Further, 
since numerous modifications and changes will readily occur to those skilled in the art, it is not 
desired to limit the invention to the exact construction and operation illustrated and described, 
and accordingly all suitable modifications and equivalents may be resorted to, falling within the 
scope of the invention. 
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